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EXECUTIVE SUMMARY 

The purpose of this report is to present the final findings and recommendations of the 
analysis performed by The Virginia State Corporation Commission Staff ("Staff) regarding the 
91 1 service outages following the June 29, 2012 storm ("June 29 Derecho"). A major focus is 
evaluating the sufficiency and extent of the corrective actions that Verizon is undertaking. The 
ultimate goal of this investigation is to help prevent such a serious and potentially life 
threatening event from occurring again. This report provides a number of recommendations to 
assist with achieving that objective. 

Early in the afternoon on June 29, 2012, a severe and destructive storm with widespread 
wind gusts of over 70 mph tracked across a large section of the Midwestern United States. The 
storm progressed into the Mid- Atlantic States in the afternoon and evening. Late in the evening, 
the storm continued to expand and impacted significant portions of Virginia, Maryland, and the 
District of Columbia with severe straight-line wind speeds reported as high as 87 mph. By the 
morning of June 30, there was an unprecedented and critical loss of 91 1 services primarily 
impacting public safety answering points ("PSAPs") and citizens in the Northern Virginia area. 

Verizon acknowledged multiple problems starting with the failure of backup generators 
to start in the Fairfax and Arlington central offices. Ultimately there was a total loss of 91 1 
telephone service to four public safety answering points ("PSAPs") (Fairfax County, Prince 
William County, Manassas, and Manassas Park) for a significant period of time. In addition, 21 
other Virginia PSAPS were impacted and experienced such problems as the failure to receive 
Automatic Location Information ("ALI") and the loss of administrative and backup telephone 
lines. 
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On September 14, 2012, the StaffReport of Preliminary Findings was filed. The primary 
focus of our investigation for that report was determining the causes of the 91 1 outages. Overall, 
we have determined that our preliminary findings have been substantiated further and 
uncontested. A list of our preliminary findings is attached to this report as Attachment 1 . 

Our continued investigation has identified a number of additional significant findings. 
These findings go beyond assessing the impact from the June 29 Derecho as they identify and 
address (1) other maintenance and equipment problems identified by Verizon, and (2) certain of 
Verizon's corrective actions and proposals, both for areas affected by the Derecho and in other 
parts of Virginia. 

Verizon has made significant progress in implementing numerous corrective actions if 
initiated shortly after the June 29 Derecho. We are pleased with Verizon's candor and direct 
approach in identifying the issues and problems. Particular recognition is given to Verizon for 
initiating power audits in central offices beyond those impacted directly by the June 29 Derecho. 
Some corrective actions were or are being implemented quickly and Verizon should continue to 
complete those initiatives as soon as possible. 

However, our investigation also raises concerns about the overall condition of Verizon's 
equipment in its Virginia offices. There is evidence of an ongoing lack of routine maintenance 
in the audited offices. This raises concerns about the conditions existing in other Verizon 
offices. It will take a concerted effort on Verizon's part to correct all the problems. This cannot 
be done overnight and likely requires oversight to ensure compliance. Our recommendations 
include details of appropriate oversight and continued monitoring. 

The 911 service outages after the June 29 Derecho put thousands of Virginia citizens at 
risk. Those outages were a direct result of Verizon's failure to perform the necessary 
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maintenance on its central office facilities, and were compounded by its inability or failure to 
monitor and respond, both internally and externally, to the outages. As the generator 
maintenance logs, audits, and other Verizon interrogatory responses reveal, Verizon has allowed 
equipment and facilities at many of its offices to deteriorate. This discovery indicates that 
Verizon should not devote its efforts solely on the sixteen 911 mission critical offices but should 
expand its corrective efforts to all its Virginia offices. 

The risk of further 91 1 or other customer service outages cannot be mitigated without 
correcting the problems at all Verizon offices in Virginia. We recognize that they cannot be 
corrected overnight. Market and technology change is resulting in industry movement away 
from the traditional copper network, particularly for voice. However, the integrity of the 
telecommunications system, regardless of technology (i.e., VoIP, wireless, FIOS, etc.), still 
depends on properly fimctioning central office equipment, transport systems, switching 
equipment, and 911 services provided by the incumbent local exchange telephone company. 

We applaud Verizon's efforts to identify and correct the underlying causes and problems, 
and are encouraged that no 91 1 -related problems arose in Virginia during and following 
Hurricane Sandy. Ensuring that testing and maintenance is performed properly and timely in all 
of Verizon's offices should go a long way in preventing a similar 911 outage as the one 
following the June 29 Derecho. In addition, Verizon has modified many of its practices to 
ensure more timely response in emergency situations. These efforts should be monitored and 
evaluated for at least some period of time (i.e., 1-3 years) to ensure that Verizon continues to 
undertake the necessary corrective actions. Finally, Verizon and the Staff should continue to 
meet and cooperate with the PSAPs to ensure their concerns are addressed. 
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INTRODUCTION 

Early in the afternoon on June 29, 2012, a severe and destructive storm with widespread 
wind gusts of over 70 mph ("June 29 Derecho") tracked across a large section of the Midwestern 
United States, The storm progressed into the Mid- Atlantic States in the afternoon and evening. 
Late in the evening, the storm continued to expand and impacted significant portions of Virginia, 
Maryland, and the District of Columbia with severe straight-line wind speeds reported as high as 
87 mph. The June 29 Derecho continued eastward during the very early morning of June 30 
affecting Delaware and New Jersey, and ultimately dissipated around 4:00 a.m. in the Atlantic 
Ocean. 

In Virginia, over one million customers lost power with many of those located in 
Northern Virginia. On the morning of June 30, Governor McDonnell declared a state of 
emergency. His announcement stated that "last night's thunderstorms caused the broadest non- 
hurricane related power outage in Virginia history." ^ 

By the morning of June 30, there was an unprecedented and critical loss of 9 1 1 services 
primarily impacting public safety answering points ("PSAPs") and citizens in the Northern 
Virginia area. Subsequently, the Virginia State Corporation Commission ("Commission") 
started receiving reports of 91 1 emergency call outages in the areas of Virginia served by 
Verizon Virginia LLC and Verizon South Inc. (collectively, "Verizon"), and unconfirmed reports 
of 91 1 service problems in the service territories of other providers. On July 3, the Commission 
issued an Order Establishing Investigation ("July 3, 2012 Order") directing its Staff ("Staff ) to 
investigate the loss of 91 1 emergency call services from the June storms. The July 3, 2012 Order 
required the Staff to report its preliminary findings by September 14, 2012, and to file a report 
with its final findings and recommendations by December 31, 2012. At the Staffs request, the 
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date for filing this report was extended to January 17, 2013. 

In addition, on July 13, 2012, the Federal Communications Commission's ("FCC") 
Public Safety and Homeland Security Bureau released a Public Notice ("PSHSB Notice") 
seeking comments on 91 1 resiliency and reliability in the wake of the June 29 Derecho in PS 
Docket No. 1 1-60. The PSHSB Notice recognized that the impact of the storm was particularly 
severe in Northern Virginia, notably in Fairfax County, parts of Prince William County, 
Manassas Park, and Manassas "where over 1 million people faced the possibility of not being 
able to call 9-1-1 successfully" and where "media reports and local government officials indicate 
that public safety answering points. ..failed as did backup systems." 1 

PRELIMINARY FINDINGS REPORT 

On September 14, 2012, the Staff Report of Preliminary Findings ("Preliminary Findings 
Report") was filed. The primary focus of our investigation for the Preliminary Findings Report 
was determining the causes of the 91 1 outages. Those findings showed that there were numerous 
and compounding errors, failures, and deficiencies on the part of Verizon that multiplied into a 
potentially catastrophic event exposing inherent weaknesses in Verizon's service and associated 
911 network design and maintenance. A list of our preliminary findings is attached to this report 
as Attachment 1. 

The Preliminary Findings Report recognized that Verizon had initiated numerous efforts 
to correct deficiencies exposed by the events surrounding the June 29 Derecho. The stated intent 
of our continued investigation was to evaluate more fully Verizon's corrective action plans going 

1 PSHSB Notice, p.l. Comments were filed in this proceeding by a number of parties, including Verizon, wireless 
carriers, Fairfax County and other localities, as well as public safety associations or entities. Reply comments were 
also filed by a number of parties. On January 10, 2013, the Public Safety and Homeland Security Bureau released a 
Report and Recommendations; "Impact of the June 2012 Derecho on Communications Networks and Services" 
("PSHSB Report and Recommendations"). http://apps.fcc.gov/ecfs/proceeding/view?name=ll-60 
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forward. Of particular concern was whether the generators in the Arlington and Fairfax offices 
had been properly maintained and tested. We were (and still are) concerned "whether such is 
indicative of a systematic deficiency throughout Verizon's network." 

STAFF INVESTIGATION 

To date, the Staff has served Verizon with five sets of interrogatories consisting of 78 
questions (many with multiple subsections). We have received and reviewed hundreds of 
documents (thousands of pages) provided by Verizon. We also met with various Verizon 
personnel on several occasions to discuss issues and responses in more detail. One meeting 
included an expansive tour of both the Arlington and Fairfax central offices on November 14, 
2012. We have participated in meetings of the Governor's Secure Commonwealth 91 1 Sub 
Panel, the National Capital Region Emergency Preparedness Council (which is part of the 
Metropolitan Washington Council of Governments ("COG")), the COG 911 Directors, and 
Fairfax County officials. We also met with members of the PSAP community in Northern 
Virginia on December 20, 2012. In addition, on November 14, 2012, COG's 
Telecommunications Network Steering Committee issued its Preliminary Report of 9-1-1 
Service Gaps During and Following the Derecho Storm on June 29, 2012. 

A major focus of our investigation for this final report was to gather as much detail as 
possible to evaluate Verizon's corrective action plans. We are seeking a better understanding of 
the depth and technical thoroughness of many of Verizon's proposed actions. An important 
aspect of our effort is determining the scope and specific schedule for the proposed changes. In 
other words, how deep into the network does Verizon intend to go (i.e., how far beyond the 

2 Preliminary Findings Report, p. 12. 

3 At its request, COG's report was filed in this proceeding. A copy of the Executive Summary of its report is 
provided as Attachment 2 to this report. 
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Fairfax and Arlington offices) with the changes and how soon will they be completed? 

Of particular concern is obtaining and evaluating the power audits conducted by Verizon 
as part of its corrective action plans. We recognize that those audits took time to perform; 
however, we did not receive all of the completed audits until December 4, 2012. We discuss 
some of our most immediate concerns from those audits later in this report. 

DISCUSSION 

Verizon has made significant progress in implementing numerous corrective actions it 
initiated shortly after the June 29 Derecho. We applaud those efforts and are pleased with 
Verizon's candor and direct approach in identifying the issues and problems. Particular 
recognition is given to Verizon for initiating power audits in central offices beyond those 
impacted directly by the June 29 Derecho. Some corrective actions were or are being 
implemented quickly and Verizon should continue to complete those initiatives as soon as 
possible. The improvements to Verizon's emergency management processes, updating manuals, 
providing additional training to its personnel, and revising generator testing procedures can be 
done quickly (and have been completed in many instances). We are encouraged by 
improvements to Verizon's documented generator maintenance/testing procedures, revisions to 
its National Power Network Operation Center ("NPNOC") procedures in identifying and 
responding to emergency conditions, and placement of readily available site specific emergency 
procedure manuals in Verizon's central offices. In addition, many of its corrective actions are 
directed (at least initially) to 16 Virginia offices that Verizon designates as its 911 mission 
critical offices. 4 

Our Preliminary Findings Report raised concerns that the generators in the Fairfax and 
4 Verizon identified offices housing E911 Selective Router Tandems as 911 mission critical. 



Arlington offices had not been properly maintained or tested and that could be indicative of a 
systematic deficiency throughout Verizon's network. We subsequently reviewed the generator 
maintenance logs in selected offices (cross section of locations and sizes) and found many 
instances of insufficient generator testing. We expect that if we look at the maintenance logs for 
other Verizon facilities in Virginia, we would see similar results. Verizon has implemented 
several corrective actions addressing generator maintenance and testing which should mitigate 
the potential for generator failures in the future. However, those practices must be adhered to in 
all Verizon facilities in Virginia, and most importantly must be continuously and routinely 
followed. We cannot forget that the 911 service outages following the June 29 Derecho would 
likely have been avoided if the generators in the Fairfax and Arlington offices had been 
maintained and operated as they were designed. 

In addition, we are encouraged by the telemetry improvements that are being undertaken 
by Verizon. It is redesigning its telemetry network to eliminate the single point of failure 
structure that presently exists. 5 Verizon's first step is to eliminate its dependency on 
Uninterruptible Power Supply ("UPS") as the primary power source by moving it to DC central 
office power. This action should help prevent the potential of a telemetry system failure 
occurring so quickly after a power outage because the UPS has a very limited reserve (i.e., 30 
minutes in the Arlington office). The very early failure of the telemetry system after the June 29 
Derecho made restoral efforts longer and more difficult. The longer term goal for redesigning 
Verizon's telemetry network in accordance with its diversity guidelines is appropriate and will 

5 Verizon utilizes a telemetry network to monitor its central offices, respond to equipment alarms, and diagnose 
network problems from various remotely located and specialized network operations centers. The present telemetry 
arrangement provides for a group of central offices to be monitored from one telemetry hub. The redesign will 
provide for the ability for each central office to be monitored from two telemetry hubs. 
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provide an even greater safety net to allow for dual monitoring of its network in emergency 
situations. 

While Verizon has been responsive to the issues identified following the June 29 
Derecho, we are concerned about the overall condition of the equipment in its Virginia offices. 
There is evidence of an ongoing lack of routine maintenance in the audited offices. This raises 
concerns about the conditions existing in other Verizon offices. Verizon has initiated efforts to 
identify and remedy the problems in its 16 mission critical offices. However, it does not have a 
plan to conduct power audits in all other Virginia offices, or to conduct transport, switching 
equipment, or operational audits in any offices. There is not a simple or quick solution to the 
lack of attention to maintenance and necessary remediation. Verizon must continue to indentify 
and remedy the deficiencies and properly maintain its facilities going forward. Moreover, 
Verizon should expand its remediation actions beyond the 91 1 mission critical offices to all other 
offices in Virginia. 

FINDINGS 6 

Overall we have determined that our preliminary findings identified in the Preliminary 
Findings Report have been substantiated further and uncontested. However, our continued 
investigation has identified a number of additional significant findings. These go beyond 
assessing the impact from the June 29 Derecho as they identify and address (1) other 
maintenance and equipment problems identified by Verizon, and (2) certain of Verizon's 
corrective actions and proposals, both for areas affected by the Derecho and in other parts of 
Virginia. These findings are not all inclusive, but provide a good foundation and cross section of 

6 All quoted material in these findings was obtained from various Verizon provided documents, including 
interrogatory responses, audits, and corrective action updates. 
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issues for evaluating the sufficiency of Verizon's corrective action plans. A copy of Verizon's 
most recent corrective action plan is included as Attachment 3. 

Our investigation has identified the following additional findings: 
Generator Maintenance, Testing, and Practices 

• A review of the 201 1 and 2012 backup generator maintenance logs prior to the June 29 
Derecho in a cross section of central offices in Virginia shows a lack of compliance with 
Verizon's maintenance and testing procedures. 

• Routine testing was not always performed and in a number of instances the testing 
duration was insufficient (i.e., a test should run the generator under load continuously for 
at least one hour every month and for five hours once a year). There were some 
instances where generators were run considerably less than an hour in a single month and 
in some cases not at all. 

• Verizon has "reviewed and refreshed its monthly and annual preventative maintenance 
requirements for generator, battery and DC plant rectifiers for all its host offices." 

• Verizon enhanced its Blackout Testing 7 procedures (as of September 2012) to include 
"failed automated controls" and "prioritized system load transfer" scenarios. Verizon 
plans to perform blackout tests for its 16 Virginia 911 network critical sites in 2013, 
scheduling each site once all the power audit findings have been addressed. It is also in 
the process of identifying which other facilities will be blackout tested in 2013. 

• Verizon has deployed standardized log books in its offices and implemented an online 
tool to track monthly and annual generator testing. 

• Verizon has formalized its generator test reporting and failure reporting procedures. 

7 Blackout testing simulates the loss of commercial power and, among other things, determines that the generators 
automatically start and office loads are automatically shifted from commercial power sources to the generators(s). 
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These procedures include an acknowledgement process for all generator run alarms by 
the responsible Network Operations Center ("NOC") and a separate operational process 
"to audit the generator run testing frequency compliance." Records will be maintained 
locally as well as by the NOC. 

• Verizon is installing a street side quick connection for a portable generator at all 91 1 
mission critical offices to be completed as follows: ten in 1Q 2013, five in 2Q 2013, and 
one by July 2013. 

• A third permanent generator will be installed in the Arlington office by 3 Q 20 1 3 . 
Telemetry 

• Verizon is redesigning its telemetry network to "include more diverse connections and 
failover (alternative) locations." 

• Verizon plans to reduce the telemetry system dependency on UPS by moving the 
telemetry equipment to a DC central office power source. This effort will provide 
additional operational duration for telemetry in the event of a commercial and generator 
power outage. 

• DC powered routers were installed at the Arlington central office on January 3, 2013 . 

• Verizon plans to redesign its telemetry network so that core routers, which receive data 
from edge routers, will have central office generator and battery back up by June 30, 
2013. 8 

• Verizon is implementing site by site remediation to move all telemetry traffic to the IP 
network. The 911 mission critical offices should be completed by December 31, 2013. 

8 "Edge routers" are deployed in each Verizon central office. They collect all of the alarms and other messages 
generated by the network equipment. Those edge routers then send that data to "core routers" - regional 
aggregation points that distribute them to the appropriate operations support systems for analysis and action by NOC 
personnel. 
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• The ongoing telemetry redesign schedule for the remaining Verizon offices (nationwide) 
is based on office size (starting from largest to smallest) with an indicated completion by 
December 3 1, 2015. This redesign will eliminate a single point of failure for the 
telemetry system allowing Verizon to maintain visibility to its central offices from an 
alternative location if necessary. 

Power Audits 

• Following the Derecho, Verizon initiated and completed backup power system audits for 
its other 911 mission critical facilities in Virginia that were not affected by the Derecho. 
It has completed backup power system audits for all of its 91 1 mission critical facilities. 

e The audits did not include assessment of switching or transport equipment. 

• The audits surveyed the overall condition of the power, grounding, and battery room 
systems at each facility and were generally "non-intrusive and visual in nature." 

• The audits also included an infrared thermographic scan of major system components. 

• Inspections of mechanical, HVAC equipment, and building conditions were also 
performed in conjunction with the power audits in the 91 1 mission critical offices. 

• The audits conducted in a few of the offices were primarily narrative in nature while the 
other audits used a survey template format with a pass/fail test. 

• The infrared thermographic scans identified variations in temperature readings (hot spots) 
in electrical equipment (i.e., connections, fuses, and circuit breakers) to diagnose 
problems (i.e., loose mechanical connections or circuit overloads) that could potentially 
cause issues (i.e., from fires to equipment failures). 

• The audits were not operational in nature and Verizon recognizes that "depending on the 
severity of any abnormalities found (if any) then a more in-depth audit of the affected 

9 



systems will have to be scheduled." We are not aware of further in-depth audits 
scheduled in any of the sixteen 911 mission critical offices. 
The audits for the 91 1 mission critical offices in Virginia identified hundreds of 
abnormalities. The underlying issues found in the audits ranged from minor to critical. 
Many of the audits show a broad range of problems from overall building deterioration to 
apparent lack of routine preventative maintenance. 

Examples of minor issues found in the various audits include low electrolyte levels in 
batteries, cracked flame arrestors on batteries, and safety equipment missing or depleted. 
The audits found instances where power equipment (i.e., battery strings, rectifiers, 
inverters) in the offices was known to be manufacturer discontinued and spare parts were 
not available on site or from the manufacturer. In many instances, the audits 
recommended replacing the manufacturer discontinued equipment. One audit stated "the 
obsolete rectifiers put the switch at risk due to lack of spare parts." 
Verizon indicates that it has access to spare parts in other locations and from third party 
vendors. In addition, Verizon retains parts from old, replaced equipment to use as spares. 
A number of the audits identified that annual Battery Run Down Tests ("RDT") had not 
been performed in some time. In one office, the audit stated the "battery plant reserve 
sizing is not sufficient to carry the load for four hours" and it was noted that the RDT had 
not been performed since 2009. 
There was evidence of roof leaks in several offices. 

The audits identified nonfunctioning and inadequate HVAC systems in some offices. For 
example, some air handling and condensing units were out of service at the time of the 
audits and the temperatures in the battery and equipment rooms exceeded acceptable 
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levels. 

• The thermographic scans identified circuit breakers and fuses in a number of offices as 
being hotter than the surrounding equipment and posing a potential fire or failure risk. 

• At one office, a brick veneer wall was falling on to the adjacent property of a U.S. post 
office. There are numerous indications of building deterioration in many offices. 

• In addition to scheduled inspections, Verizon will conduct backup power system audits in 
facilities other than 911 mission critical offices based on "field requests or direction from 
the engineering teams." For example, items noticed in other inspections and activities in 
an office may trigger a field request for a power audit. 

• The audits in several offices identified instances where power rectifiers were not 
operational, needed to be repaired, or were improperly located. 

• Verizon has addressed and repaired many of the problems identified in the audits and has 
stated it is committed to rectifying all problems indentified therein. 

Other Initiatives 

• Outside the power audit process, Verizon inspected during August and September of 
2012 all its Virginia central offices under its annual "Neat, Clean, Safe and Reliable" 
central office inspection program. 

• According to Verizon, this program has resulted in hundreds of work action items in over 
300 Virginia offices. 

• The work action items are categorized and prioritized into three groups: Network 
Reliability, Safety/Security, and Neat Clean. 

• Verizon has completed the work on many of these work actions and plans to address all 
items identified in the Virginia offices. 
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• Verizon accelerated its "Original Equipment Manufacturer Preventative Maintenance" 
program that it had planned for 2013 into 4Q 2012. Under this program, the original 
equipment manufacturer ("OEM"), or a qualified OEM vendor, performs an annual 
checklist of maintenance items. During 4Q 2012, Verizon completed this program in 222 
offices nationwide, including 29 in Virginia. 

• Verizon has deployed additional diversity in its long haul network and has retired some 
specific transport systems that were at the end of their life. 

• Verizon has completed a bar code inventory of all plug-in equipment for is legacy local 
network. 

Emergency Practices and Procedures 

• Verizon revised its National Power Network Operation Center ("NPNOC") 
storm/emergency handling procedures. 

• The revised NPNOC procedures were expanded to address "unplanned" significant 
natural events or storms. The NPNOC will declare a "storm/emergency event" during an 
unplanned event when there are five or more battery on discharge ("BOD") alarms in the 
same geographic area. 

• When a storm/emergency event is declared (from either a forecasted or unplanned event), 
the NPNOC will contact the Global Event Network Management Center ("GENMC") 
and the GENMC will then convene a conference bridge to outline a strategy plan. 

• The NPNOC will no longer suspend notifications of BOD alarms to the field during 
storm emergencies. All escalations to the dispatch centers and field power managers will 
continue during such emergencies. 

• Verizon is adding an additional power technician to its evening shift in Northern 
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Virginia. In addition, "Verizon has modified its work schedule to designate Arlington 
and Fairfax offices as the primary reporting location...." 

• Verizon placed a readily available copy (or copies) of a site specific "Central Office 
Emergency Procedure Manual" in its 91 1 mission critical offices that identifies and maps 
the power and generator procedures for the specific office. 

• Verizon has centralized its emergency activation processes with its Business Continuity 
and Emergency Management (BCEM) organization to expedite prompt responses to 
unforeseen emergencies. 

911/PSAP 

» As part of its corrective actions, Verizon has agreed to implement a number of initiatives 
requested by the PSAPs. 

• Verizon is developing a centralized system to access the PSAP's 91 1 infrastructure 
information and is working with the PSAPs to make this available to them. 

• Verizon is working with PSAPs in Northern Virginia to recommend design changes to 
improve diversity and infrastructure inventory. 

« All Northern Virginia PSAPs have been provided with design change recommendations. 

• Verizon plans to complete its diversity review for all PSAPs in Virginia by year end 
2013. 

• Verizon plans to conduct a drill or exercise with each requesting jurisdiction on a 
semiannual basis that models potential or actual 911 outages. In early December, 
Verizon conducted a pilot exercise with the City of Norfolk. 

• The PSAPs appear to be encouraged by the actions being undertaken by Verizon's 
current operational management but still have concerns about the sustainability, 
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adequacy, and timing of the proposed corrective actions. 



RECOMMENDATIONS 

Our goal for this investigation is to help prevent such a seriously and potentially life 
threatening event from occurring again. We recognize that there are no "absolutes" to prevent 
another 911 service outage. However, the 91 1 outage following the June 29 Derecho in 
Northern Virginia should not have occurred and was avoidable if Verizon had properly 
maintained the generators in the Arlington and Fairfax offices. Nonetheless, the 91 1 outage and 
subsequent Verizon investigation exposed numerous deficiencies and weaknesses inherent in its 
procedures, processes, and central offices. 

Our investigation shows that Verizon has resolved many problems, and is initiating 
actions to correct additional deficiencies. However, it will take a concerted effort on Verizon's 
part to correct all the problems. This cannot be done overnight and likely requires oversight to 
ensure compliance. Therefore, we recommend the following: 

• This docket should remain open. 

• Verizon should be required to update and file quarterly corrective action progress reports 
with the Commission. 

• Verizon should correct all deficiencies and implement all recommendations identified in 
its power audits. 

• Verizon should meet quarterly with the Staff to provide additional details, schedules, 
budgets, and updates on its corrective actions, audits, inspections, and other initiatives 
intended to correct its deficiencies in Virginia. 

• Verizon should continue to meet and cooperate with the PSAPs to ensure their concerns 
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are addressed. 

By the end of 1Q 2013, Verizon should develop and review with the Staff a schedule to 
conduct audits (including power, mechanical, and HVAC equipment) in all remaining 
Virginia offices. Verizon should permit the Staff to monitor any audit as it is conducted. 
Recognizing the time required to complete the audits, at a minimum, batteries should be 
inspected and tested in all Virginia locations by the end of 2Q 2013. 
Verizon should provide the Staff with copies quarterly of any additional or revised power 
audits conducted for offices in Virginia. 

Verizon should provide the Staff with any plans to conduct additional inspections or 
audits for switching and/or transport equipment and operational audits in Virginia. 
Copies of the results from any such inspections and audits should be provided to the 
Staff on a quarterly basis. 

Verizon should establish a plan to address the availability and sufficiency of spare parts 
for manufacturer discontinued equipment. 

The Staff should continue to communicate and meet with PSAPs and the 91 1 
community. 

Verizon should maintain and update a complete inventory of its 91 1 service 
infrastructure. 

Verizon should provide a quarterly report to the Staff identifying any problems found in 
the monthly testing of generators in offices in Virginia. The report should indentify the 
office and the corrective action undertaken and include applicable dates. 
The Staff should file an annual status report with the Commission that includes 
recommendations on continuing the various requirements on Verizon and/or 
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recommendations on any changes or additions to such. 
• The Staff should evaluate the FCC Public Safety and Homeland Security Bureau's 
Report and Recommendations released on January 10, 20 13, 9 and advise the 
Commission of any additional recommendations we may determine are warranted based 
on that report. 

CONCLUSION 

The 911 service outages after the June 29 Derecho put thousands of Virginia citizens at 
risk. Those outages were a direct result of Verizon's failure to perform the necessary 
maintenance on its central office facilities, and were compounded by its inability or failure to 
monitor and respond, both internally and externally, to the outages. As the generator 
maintenance logs, audits, and other Verizon interrogatory responses reveal, Verizon has allowed 
equipment and facilities at many of its offices to deteriorate. This discovery indicates that 
Verizon should not devote its efforts solely on the sixteen 911 mission critical offices but should 
expand its corrective efforts to all its Virginia offices. 

The risk of further 91 1 or other customer service outages cannot be mitigated without 
correcting the problems at all Verizon offices in Virginia. We recognize that they cannot be 
corrected overnight. Market and technology change is resulting in industry movement away 
from the traditional copper network, particularly for voice. However, the integrity of the 
telecommunications system, regardless of technology (i.e., VoIP, wireless, FIOS, etc.), still 
depends on properly functioning central office equipment, transport systems, switching 
equipment, and 911 services provided by the incumbent local exchange telephone company. 

We applaud Verizon's efforts to identify and correct the underlying causes and problems, 

9 On January 10, 2013, FCC Chairman Genachowski announced plans to strengthen the reliability and resiliency of 
the nation's 911 services during major disasters. A copy of that public announcement is attached to this report as 
Attachment 4. 
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and are encouraged that no 91 1 -related problems arose in Virginia during and following 
Hurricane Sandy. 10 Ensuring that testing and maintenance is performed properly and timely in 
all of Verizon's offices should go a long way in preventing a similar 91 1 outage as the one 
following the June 29 Derecho. La addition, Verizon has modified many of its practices to 
ensure more timely response in emergency situations. These efforts should be monitored and 
evaluated for at least some period of time (i.e., 1-3 years) to ensure that Verizon continues to 
undertake the necessary corrective action. 



While not of the magnitude or seriousness of the June 29 Derecho in Virginia, Hurricane Sandy caused 
commercial power loss to three Verizon central offices in Virginia, including Arlington, and the generators 
automatically started in all. 
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Attachment 1 



SEPTEMBER 14, 2012 STAFF REPORT 
PRELIMINARY FINDINGS 



Commission Rule 20V AC 5-425-40 A 1 requires a LEC providing 911 service to 
"design, construct, maintain, and operate its facilities to minimize interruptions to E-91 1 
services." 

Verizon was the only LEC in Virginia that experienced significant 911 service problems 
following the June 29 Derecho. 

The total loss of 91 1 capabilities to the Prince William County, Fairfax County, 
Manassas, and Manassas Park PSAPs was an extremely serious event and it is very 
fortunate that there were not catastrophic consequences to any citizens in Northern 
Virginia 

The Prince William County, Fairfax County, Manassas, and Manassas Park PSAPs were 
fully prepared to respond to the June 29 Derecho and were not responsible for the 91 1 
service failures. 

The cause of the 91 1 service outages in Northern Virginia from the June 29 Derecho 
began with the failure of two backup generators that did not start automatically when 
commercial power was lost. Specifically, a generator in each of Verizon's Arlington and 
Fairfax central offices did not start. 

A review of the maintenance logs for the backup generators in the Arlington and Fairfax 
central offices shows a lack of compliance with Verizon's maintenance and testing 
procedures. 

The generator that failed to start in the Arlington office did not start during routine testing 
conducted two days before the June 29 Derecho. The maintenance log indicated that 



work to the generator was needed. 

• A total of nine generators (out of 136) failed to operate properly during the commercial 
power outages from the June 29 Derecho in Verizon's Mid- Atlantic region. 

• The scope of 9 1 1 problems went well beyond the calling areas served by the Arlington 
and Fairfax central offices. 

• ALI is an important component of 91 1 service. The lack of delivery of ALI to many 
PSAPs could have put citizens across Virginia at greater risk. 

• The initial battery on discharge ("BOD") alarm 1 for the Fairfax central office was sent to 
the National Power Network Operation Center ("NPNOC") at 10:29 p.m. on June 29, 
2012 when the one generator failed to start. Under Verizon's procedures, any BOD 
alarm should have been seen as a critical power alarm requiring immediate action. 
However, according to Verizon, this alarm was incorrectly categorized as a major power 
alarm condition when sent to the NPNOC. 

• The Regional Network Center ("RNC") received a repair ticket (identified as a major 
alarm as mentioned above) from the NPNOC for the Fairfax central office at 10:32 p.m. 
on June 29, 2012. At that time, and on the morning of June 30, the RNC was only 
working critical alarms and a power technician was not dispatched to the office until 
after the backup batteries had drained completely. 

• The telemetry system (alarm monitoring) in the Arlington central office was only 
supported by the Uninterruptable Power Supply ("UPS") (i.e., battery power source) 



BOD or battery on discharge usually indicates one of two conditions. One is an all rectifier failure with or 
without a generator or commercial power failure, and second is a commercial power failure with generator failure. 
In each situation the office batteries are being depleted and the alarm condition is a CRITICAL indicator that 
network service is in jeopardy. 



which was designed with a 30 minute reserve. The UPS failed at 1 1 :23 p.m. on June 29, 
2012. 

The very early failure of the telemetry system resulted in Verizon being unable to receive 
further alarms and remotely access its switches to monitor, test, or reroute traffic to 34 
sites in the area. Verizon's inability to monitor its facilities and network in the Northern 
Virginia area significantly impacted the restoral process from the June 29 Derecho. 
The delay in identifying and repairing the critical conditions in the Fairfax and Arlington 
offices resulted in unnecessary damage to Verizon's network and extended the 91 1 
problems and outages. There were hundreds of damaged or impacted pieces of 
equipment in those two offices (i.e., circuit cards and digital cross connects). 
The loss of the transport systems in the Arlington and Fairfax central offices was 
profound and collectively resulted in 17 switches becoming SS7 isolated, and therefore 
incapable of completing (originating or terminating) any interoffice local, long distance, 
or 91 1 emergency calls. The loss of those transport systems was also responsible for the 
loss of ALI to the PSAPs. 

Verizon did not activate its emergency Area Control Center located in Maryland until 10 
a.m. on June 30, 2012. 

Verizon did not always provide sufficient, accurate, or timely communications to the 
affected PSAPs regarding its 91 1 problems and outages following the June 29 Derecho. 
Some battery reserves supporting major equipment systems in the Arlington (other than 
telemetry) and Fairfax central offices were depleted within approximately 3 to 5 hours. 
In addition, some equipment in those offices failed even before the batteries exhausted 
because of sensitivity to low voltage conditions. 

In many instances, Verizon's workforce was not timely dispatched, prepared, or trained 



to recognize or correct the critical conditions from the June 29 Derecho. 
• Verizon is making progress in implementing its corrective action plan, however, at this 
time, not all items have been fully defined or timelines determined. 



Attachment 2 



PRELIMINARY REPORT of 9-1-1 
SERVICE GAPS DURING and 
FOLLOWING the DERECHO STORM 
on JUNE 29, 2012 



Metropolitan Washington Council of Governments 
9-1-1 Telecommunications Network Steering Group 



November 14, 2012 



EXECUTIVE SUMMARY 

The 9-1-1 Emergency Call System is the vital link to public safety assistance across the country, providing 
access to police, fire and emergency medical services. Residents and visitors in cities, towns and rural 
communities are confident that accessing 9-1-1 will result in saving lives and property. It is the public's 
expectation that the responsibility of public safety and local and state government officials is to assure 
that the fees and charges assessed for 9-1-1 service are used to provide continuous and reliable public 
safety service. The National Capital Region (NCR), as the nation's capital and a major urban center, must 
have a reliable 9-1-1 system. 



Late on the evening of Friday, June 29, 2012, a severe storm (Derecho) hit the Mid-Atlantic region with 
unusually intense straight-line winds. The storm caused widespread commercial electric power and 
communications outages in Washington D.C., Virginia, Maryland and additional states. At approximately 
7:30 AM on Saturday, June 30, 2012, the 9-1-1 centers in Fairfax County, Prince William County, Cities of 
Manassas and Manassas Park experienced a complete failure of Verizon's 9-1-1 and telephone service. 
Three additional 9-1-1 centers, Arlington County, City of Alexandria and Loudoun County experienced a 
partial failure of these services. Verizon's restoration of 9-1-1 service began at approximately noon on 
Saturday, but some of the issues continued for over 5 days until July 4 th , 2012. 

Metropolitan Washington Councils of Governments Response 

While the states and federal government regulate telecommunication utilities, 9-1-1 connects people in 
need with local governments. Thus, the failure of this system as a result of the Derecho became an 
issue of great concern to the Metropolitan Washington Council of Governments (COG), an association of 
22 local governments that represent over 5 million residents. In addition, there have been previous 
issues with 9-1-1 service, that have been brought to Verizon's attention as indicated in a letter to 
Verizon from COG dated July 21, 2011. 

On July 11, 2012, at its regular meeting, COG Board of Directors unanimously adopted R36-2012 
Resolution to Encourage Steps to Address Verizon 9-1-1 Service Gaps During and Following the Derecho 
on June 29, 2012 which included the five items below 

• Cause of Verizon's 9-1-1 failure; 

• Existing redundancy and backup capabilities; 

• Vulnerability of newer technologies that required battery or back-up power, including home and 
business service; 
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• Opportunities for COG localities to influence and strengthen regulatory oversight and remedies 
at the state and federal levels; 

• Verizon's communication and messaging to the public and local emergency response officials 
concerning 9-1-1 services 

COG formed a task force of 9-1-1 Center Directors and other interested parties to address the five items 
in the resolution. The following are the preliminary findings of the task force. 

1. Cause of Verizon's 9-1-1 Failure 

The loss of commercial power and the subsequent failure of one of two backup generators in each of 
Verizon's Arlington and Fairfax Central Offices (CO) were the predominant causes of the 9-1-1 service 
outages. 

• The Derecho impact on the electrical infrastructure caused the loss of commercial power to the 
Verizon facilities located in Arlington and Fairfax, Virginia and elsewhere. 

• The back-up generator, in the Fairfax CO, that supported 9-1-1 systems did not start 

• In addition, the back-up generator in the Arlington CO, that supports Verizon's ability to view, 
monitor and identify problems in its network, did not start. 

• Verizon had failed to identify or resolve previously identified maintenance issues with these 
generators; air in the fuel lines or faulty automatic fail-over switches, incorrect log entries and 
corrective action. 

• Verizon's technician dispatched to Fairfax CO, on the morning of Saturday, June 30, 2012, did 
not realize, and took several hours to identify, that the generator supporting the 9-1-1 
infrastructure was not operating. The delay allowed the batteries to drain resulting in the loss of 
9-1-1 services. 

• Both the Verizon Arlington and Fairfax facilities were supported by back-up battery power, but 
these batteries drained. 

• Verizon failed to successfully implement any mitigating action to restore these two generators 
prior to the battery back-ups expiration. 

• Once the battery supplies were exhausted both the ability to view and identify problems and 9- 
1-1 systems at the Verizon Arlington and Fairfax facilities failed. 

• In addition, damage and failure of other 9-1-1 supporting systems within the Verizon network 
and infrastructure, such as the ability to receive the callers location, severely contributed to the 
9-1-1 outage. However, these were largely cascading effects related to the loss of adequate 
backup power in Arlington and Fairfax COs. 



2. Existing Redundancy and Backup Capabilities 

This report addresses the three major components of 9-1-1 services from both the 9-1-1 Service 
Provider (Verizon) and Public Safety Agencies perspective to include Power, Network and 9-1-1 center 
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Power 



Verizon and other telecommunications providers and many of the 9-1-1 centers have designed and 
implemented backup power systems in most of their critical facilities that include generators and 
backup battery supplies in case of commercial power failure. In some cases they have worked with the 
local power companies to implement dual power sources from separate power feeds. 

In the case of the Derecho on June 29, 2012, the 9-1-1 center and other telecommunications providers' 
backup power systems generally operated as designed and continued to provide required power until 
commercial power was restored. The generator issues experienced by Verizon, however, had significant 
impacts. 

• Network 

Verizon's network to provide 9-1-1 services includes multiple levels of diversity and redundancy, as well 
as back-up power in critical facilities, to optimize resiliency during a crisis. 

• 9-1-1 Centers 

Most of the critical systems and facilities, including servers, workstations, and databases, within the 9-1- 
1 centers have redundant components that are designed to provide continuous service and mitigate any 
downtime. In addition, many of the 9-1-1 centers have backup locations where calls can be routed in the 
case of major outages or the loss of the primary 9-1-1 center. In the case of the Derecho event, many of 
the backup 9-1-1 centers' services were provided through the Verizon Arlington and Fairfax locations, 
and thus were also unable to receive emergency calls. 

3. Vulnerability of Newer Technologies that Require Battery or Back-Up Pow er. Including Home 
and Business Service 

For many decades, power for traditional telephone service for most residences and small businesses 
was supplied via the hard wire connection through the telephone lines and therefore the loss of 
commercial power often did not result in the loss of dial tone or telephone service. Today, the 
widespread use of cordless phones which depend on commercial and limited battery power, results in 
the loss of telephone service during power outages. 

Certain more recent technologies such as Voice over Internet Protocol (VoIP) or Standard Internet 
Protocol (SIP) rely on a modem or router located on premise or within a computer. With the use of 
these technologies, the loss of power causes the loss of telephone service and access to 9-1-1 once the 
back-up battery contained within the equipment, drains. 

Some commercial or business telephone systems, primarily for smaller businesses, might also require 
power to operate properly. In addition, mobile telephone service, when a high volume of calls are being 
attempted into the mobile network at the same time, can cause network congestion and/or blockage. 
Also, the loss or failure of the mobile infrastructure, such as physical damage to cell sites, or network 
connectivity, can impact the ability to make and receive mobile calls and therefore access to 9-1-1. 
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4. Opportunities for COG Localities to Influence and Strengthen Regulatory Oversight and 



Remedies at the State and Federal Levels 



At the time of this preliminary report there are five proceedings related to the Derecho and its impact 
on 9-1-1 services. 

It is anticipated that reports will be issued by these various groups which will be incorporated into the 
final version of this report. 

Within the proceedings, listed below, authorities in the COG region should encourage the adoption of 
new rules that would require Verizon, and other 9-1-1 service providers to adhere to high standards of 
operation to better ensure and support 9-1-1 service or face penalties. 

1. Virginia SCC Case No. PUC-2012-00042 
http://www.scc.virginia.gov/case/PublicComments.aspx 

2. FCC PS Docket No. 11-60 

http://www.fcc.gov/document/pshsb-seeks-comment-effects-derecho-storm-communications ) 

3. Virginia Secure Commonwealth Panel - 9-1-1 Sub Panel 

4. Maryland Public Service Commission Case No. 9298 

http://webapp.psc.state.md.us/lntranet/Casenum/CaseAction new.cfm?CaseNumber=9298 

5. Maryland Emergency Number Systems Board (ENSB) Inquiry 

5. Verizon's Communication and Messaging to the Public and Local Emergency Response Officials 
Concerning 9-1-1 Services 

Public messaging was needed not only from the public information officers (PIOs) supporting the 9-1-1 
centers, but from the utility itself. As part of the overall system of disseminating information to the 
public, Verizon needed to be part of the many voices with the common message that the 9-1-1 system 
was down. Verizon should have pointed to the local officials' guidance on what the public should do in 
case of an emergency, especially during this event, when everyone was challenged by lack of electricity, 
phones and connectivity. Officials needed a more robust public messaging response on Verizon's part to 
complement local government efforts. In these reports, Verizon states it is mobilizing a more robust 
emergency response communications process to ensure that media outlets and other channels are 
provided relevant information on a timely basis. 

Verizon's first responsibility, in a service interruption, is to notify the 9-1-1 center. Then in its role as a 
local utility, in cooperation with local government, Verizon has the responsibility in providing enhanced 
customer service, to inform the public of 9-1-1 interruptions. This should include dissemination of 
information about the extent of the problem and when it will be resolved. PIOs and 9-1-1 centers 
should remain the primary source of guidance to the public during an emergency. 
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Recommendations 

On July 19, 2012 the Northern Virginia 9-1-1 Directors (City of Alexandria, Arlington County, Fairfax 
County, Prince William County and Stafford County), and subsequently all of the 9-1-1 Directors in the 
COG Region, concurred on five recommendations which were accepted by Verizon and are in various 
stages of completion. 

1. Adoption of the National Incident Management System (NIMS) Model ( www.fema. gov/national- 
incident-management-svstem ) 

2. 9-1-1 Interruption Notification 

3. Semi-annual 9-1-1 Outage Drill 

4. Monthly update of contact list 

5. Verizon Emergency Operations Center (EOC) Representative 

In addition to the recommendations of COG 9-1-1 Directors released in the aftermath of the Derecho 
event, which Verizon should continue to implement, there are several other recommendations from 
COG 9-1-1 Telecommunications Network Steering Group and 9-1-1 Directors that should be considered 
and are outlined below. 

1. Federal and State Regulatory Authorities should strongly encourage Verizon and other 9-1-1 
service providers to perform a comprehensive independent audit of the entire infrastructure, 
processes and procedures that support 9-1-1 service and related systems, to assure the 
reliability and continuity of 9-1-1 service under any circumstance. Based on the results of these 
audits, comprehensive plans and strategies should be developed to immediately resolve any 
findings. The results of these audits and resolution plans should be made available to the 9-1-1 
stakeholders. 

2. It is highly recommended, that Verizon and other 9-1-1 service providers should provide subject 
matter expertise and make recommendations to the 9-1-1 centers and their stakeholders to 
assure reliability and continuity of 9-1-1 service. This should include, but not be limited to, 
network redundancy, 9-1-1 center equipment and systems, and best practices and procedures. 

3. It is critical, that Verizon review their communications and public notification plans with each 9- 
1-1 center's communicators and/or Public Information Officers (PIO) regarding the 
dissemination of emergency messages (using both traditional and social media) to the public 
during 9-1-1 outages and update as needed. This process should also explore alternative 
methods to communicate with the public in case of widespread power and telephone outages. 
Verizon should coordinate with National Capital Region communicators/PIOs during any future 
outages, to inform and keep the public updated, and amplify the 9-1-1 center-specific public 
messages and information. 

4. Verizon should keep the public informed of any service issues, the extent of the outage and time 
for resolution. 

5. Federal and State Regulatory Authorities should evaluate the steps and actions of Verizon, 
related to this event, and the above audits, to ensure Verizon has adequately resolved all issues 
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and continues to improve their processes and infrastructure to ensure reliability and continuity 
of 9-1-1 service, 

6. COG members and localities should work with their State and Federal regulatory authorities and 
Legislators, as needed, to assure, through proper oversight, best practices and procedures by 
establishing service level agreements to ensure reliability and continuity of 9-1-1 service. 

7. It is recommended that there be further investigation by State and Federal Regulators, on 
whether the 9-1-1 supporting infrastructure of other telecommunications providers other than 
Verizon, was also impacted by the Derecho. As an example, AT&T Wireless in their comments to 
FCC PS Docket No. 11-60, indicated there was some impact to their infrastructure during and 
after the Derecho. 

By all indications during this event, the systems and processes in place by the public safety agencies in 
the COG region, operated as designed, and the 9-1-1 centers were fully prepared to provide service to 
the public. But, there are some items, which need to be considered, by local and state government 
officials, to ensure future reliability and continuity of 9-1-1 services which are as follows. 

1. State and local 9-1-1 authorities should be encouraged to perform a full assessment of their 
current 9-1-1 systems and operations to assure reliability and continuity of 9-1-1 service. 

2. It is recommended that State and Federal regulatory authorities, review current laws and 
regulations related to 9-1-1 service, to assure it places emphasis and favors public safety versus 
the 9-1-1 service providers or telecommunications providers. The interest of the public and 
public safety should come first over the interest of commercial providers. 

3.. State and local 9-1-1 authorities should work with their Legislators to ensure that the funding 
required to support the current 9-1-1 services and future Next Generation 9-1-1 are adequate 
and available, and that the fees and funds collected from the citizens of their States for 9-1-1 
services are dedicated and used solely for the purpose as intended for the implementation, 
operation and maintenance of 9-1-1 emergency telephone services as required by the 
Enhance911 Act of 2004(Pub. Law 108-494). In addition, the fees collected should be fairly and 
equally distributed to the 9-1-1 authorities. 

Outstanding Issues 

There appears to be no standards for 9-1-1 service providers to adhere to public safety grade 
requirements for backup power related to the systems that support 9-1-1 services. 

Next Steps 

1. COG should formalize a committee of 9-1-1 Directors that can address specific issues related to 
9-1-1 emergency telecommunications service for the NCR 

2. COG, with the assistance of the 9-1-1 authorities, should take the lead to work cooperatively in 
the development of a multi-year 9-1-1 strategic plan to include Next Generation 9-1-1 
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Conclusion 



The Derecho's impact on 9-1-1 services and the ensuing public and industry reaction has been one of 
the most significant events in the history of enhanced 9-1-1 services in the United States. It is 
conclusive that there were many areas in which Verizon could have performed better related to their 
initial response to the issues the Derecho storm created. Questions still remain about the current 
reliability, age and condition of the Verizon infrastructure local governments rely on to provide life- 
saving 9-1-1 public services. 

In the aftermath of the storm, Verizon has taken steps to address the issues of June 29, 2012. Verizon, 
however, has a responsibility to follow-up on the additional recommendations of this and other 
recommendations made to the FCC hearings. Verizon must continue to evaluate their ongoing 
operations, processes and best practices to mitigate the impacts of this type should it happen again. 
There is much Verizon must do to regain the confidence of the public safety community and citizens that 
their part in providing vital 9-1-1 communications service is highly reliable and sufficiently redundant on 
an ongoing basis. 

There were no identifiable issues for the 9-1-1 centers during this event and all of their systems 
operated as designed. The public safety community, however, must also shoulder the responsibility to 
determine where improvements can be made and make plans for continuous improvements to meet 
new operational and technology challenges. State and federal government officials need to provide 
resources to the public safety community, and proper oversight, to allow the technology and human 
resources that are necessary to support the operations of the current 9-1-1 services as well as Next 
Generation 9-1-1 services. 
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Attachment 3 



Verizon, 911 Service 
and the Derecho 

Moving Forward 
Corrective Actions Update 



January 8, 2013 

Maureen Davis 
Vice President Network Operations 
MidAtlantic 



verizon 



Power 



The specific cause of the Northern Virginia 91 1 disruptions was the failure 
of one of two back-up generators to start in Arlington and Fairfax 
following the loss of commercial power. These problems are fixed 



Issue 



Arlington Back-up Power 



Action Plan 



Install new start batteries on Generator 1 
Complete Generator 2 repairs 

Complete full load transfer test (pending battery tests 
and run down testing) 

Complete fuel system repairs 

Update manual generator-starting procedure 

Install a new UPS on the Generator 2 Auto Transfer 
Switch (this solves the Generator 2 start failure) 
Install a permanent Auto Transfer Switch power 
source via the inverter power plant that is tied to DC 
power plant batteries. 



Status 



Complete 



Complete 



verizon 



Power (cont'd) 



Verizon's investigation revealed significant opportunities for improvement 
to ensure that best practices are followed and lessons learned are 
applied throughout Verizon's service territory. 



Issue 



Generator system failures were different in 
each location. While the specific failures 
have been repaired, we have extended our 
review across the footprint to identify and 
address potential vulnerabilities. 



Action Plan 



Conduct back-up power system audits in the 
mission-critical Verizon facilities supporting 911 in 
Virginia, Maryland and Washington, DC. 

These audits include ensuring the proper 
categorization of power alarms, as the 
investigation revealed that an alarm from Fairfax 
before the loss of network monitoring was mis- 
categorized and thus placed lower on the priority 
list. 

Institute any corrective measures identified in 
those power audits. 



Where multiple generators are present, we will 
institute automated controls to prioritize system 
loads so that critical elements (e.g., network 
monitoring) stay up or are restored first in case 
one of the two generators fails. 



Status 



Completed for VA, MD and DC sites- 
10/17/12 



• Power alarms tested for 3 most 
critical conditions for all Verizon 
central offices: 12/5/12 

• Remediation for such 
conditions is 98% complete 

• Corrective actions for audit findings in 
progress. Most items will complete 
by 3/31/13; some items require longer 
time frames and are scheduled for 
2Q13 and 3Q13. 

• Arlington audit concluded that 
automated load shed not optimal 
solution; a 3 rd permanent generator 
will be installed 3Q13. Existing 
portable generator to remain on site 
and connected until permanent one 
installed. 



verizon 



Issue 



Emergency Power Practices and 
Procedures 



Verizon will improve its speed of 
restoration of power, moving to manual 
starts where necessary without delay, 
prioritizing power to key network 
equipment (e.g., 911, monitoring systems) 
in multi-generator configurations, and 
improving its deployment of mobile 
generators. 



Power (cont'd) 



Action Plan Status 



Implement site-specific back-up power system 
procedures at critical facilities to ensure real-time 
on-site accurate identification of power loss 
anywhere in the facility. 

Create site-specific manual generator start and 
transfer procedures, including prioritized system 
loads, to ensure a rapid start in the case of failure 
of automated starting systems. 
Enhance critical facility "Black Out" testing. We test 
our back-up power systems regularly, but will now 
include "failed automated controls" and "prioritized 
system load transfer" scenarios. 

Improve training and testing compliance. Our 
investigation revealed that the generator in 
Arlington had been tested just prior to the Derecho, 
failed to start, and required service, but that 
procedures weren't followed that would have 
ensured speedy correction of those service issues 
and/or faster restoration of the office. 



Potomac-Complete 
Footprint-1Q2013 

Potomac- Complete 
Footprint- 201 3 



Field Blackout Tests 
1Q13 



Complete 



v£izon Emergency Management Processes 



More robust visibility into our network and crisis management processes 
will improve coordination and communication with PSAPs and other 
government/local officials. 



Issue 



Verizon has a standard practice of internal 
mobilization based on actual or potential service 
impacts. These are triggered by alarms. The loss of 
visibility into our network prevented us from 
receiving these alarms and delayed our response. 



Action Plan 



We have enhanced our event criteria and procedures for 
notification and mobilization to trigger activity more quickly 
when batteries are activated or when network monitoring is 
lost in multiple offices in a geographic area. 




Rapid identification of emergencies and transition to 
Emergency Management. The Derecho was initially 
treated more like an internal network problem than 
like a hurricane-type problem, and this affected 
incident management. 



Emergency Management has been centralized and 
enhanced so that all emergencies, including network 
emergencies, are managed by Verizon's National 
Emergency Coordination Center (NECC), which utilizes the 
National Incident Management System (NIMS) principles 
as published by the U.S. Department of Homeland 
Security. 



Complete 



Verizon Network 



Telemetry systems allow Verizon to receive alarms, monitor its network, 
identify the cause and location of problems, and repair them rapidly. 



Issue 



Creating diverse connectivity and alternative 
telemetry sites will provide greater resiliency in 
crises. It will also improve the effectiveness of real- 
time communications with PSAPs. 

• This initiative will enhance visibility into the 911 
network. For example, our investigation 
revealed that the Eastern portion of Loudoun 
County could not reach the County's PSAP for 
several hours on June 30, but the loss of 
telemetry impeded effective communication with 
the PSAP on the issue. 



Action Plan 



Redesign the telemetry network to include more diverse 
connections and failover (alternative) locations. 

Diversity guidelines for telemetry network published 8/15/12; 
implementation plan approved. 

Replaced major hub routers in Arlington with DC powered 
routers to extend telemetry life to that of office battery power 
-completed 1/3/13. 

Redesign telemetry edge routers to a core router that has 
generator and DC battery back-up. Virginia offices to 
complete by 6/30/13. 

Site-by-site remediation to upgrade the edge routers and 
move all telemetry traffic to the IP network across the entire 
Verizon Telecom footprint. All E911 tandem offices 
throughout that footprint will be finished by 12/31/13. 



verizon 



911/PSAPs 



Verizon's analysis of the network impacts following the Derecho has 
identified areas for improvement with specific PSAP configurations, 
especially involving ALI and trunk diversity. Verizon will work directly 
with the specific PSAP partners to make those improvements. 



I 

Issue 


Action Plan 


Status 


Opportunities for improved diversity on 
PSAP trunking and ALI links. Conduct 
network design review for all Maryland 
and Virginia PSAPs. 


• Review PSAP trunking and ALI links 
for diversity 

• Work with local Engineering and 
Operations team to remediate issues 
identified. 


• Review completed for affected PSAPs 
in Northern Virginia 7/31/12. 

• Virginia redesign recommendations 
ready for review; scheduled with 
PSAPs as available. 

• Maryland reviews completed by 
9/30/1 2 and reviewed with PSAPs in 
October and November. Redesign 
recommendations to follow. 


A centrally inventoried 911 Infrastructure 
will facilitate trouble-shooting and 
improve restoration times. 


• Develop a means to implement and 
maintain an inventory for E91 1 
Infrastructure. 


• Technical service managers to retain 
all currently developed network routing 
maps. 

• Network routing maps will be 
integrated into new ticketing systems 
to allow for faster response and 
facilitate trouble-shooting and 
restoration. 



verizon 



Communications 



The 91 1 Directors of the City of Alexandria, and the Counties of 
Arlington, Fairfax, Loudoun, Prince William and Stafford have 
recommended that Verizon adopt five steps to improve communications 
and crisis response. Verizon has adopted those concepts. 



Recommendation 



Action Plan 



Verizon adopt, embrace, instruct, train and utilize 
the National Incident Management System (NIMS) 
model, to address and mitigate any and all 
significant events/incidents impacting providing 
911 service to the aforementioned jurisdictions. 



Complete 



Verizon employs an "all hazards approach" to its 
Business Continuity, Disaster Recovery, Facility 
Preparedness and Emergency Management programs. 
These are essential to the protection of its employees, 
critical business processes and structural facilities 
located around the globe. 

Verizon's National Emergency Coordination Center 
(NECC) process utilizes the National Incident 
Management System (NIMS) principles as published by 
the U.S. Department of Homeland Security. Verizon 
offers internal training and orientation courses on its 
NECC processes, and an Introduction to the National 
Incident Management System. 
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Communications (cont'd) 



Recommendation 



Verizon obtain and utilize a Reverse 91 1® type system 
to notify, via voice and text, those persons identified by 
the above jurisdictions, as soon it is known or 
suspected by Verizon that there is or may be an 
interruption of 9-1-1 service to any or all of the above 
jurisdictions. The immediately transmitted voice and 
text message should contain, in plain language, the 
nature of the problem, current or potential impact of the 
problem, what Verizon is doing to address the problem, 
recommend actions the impacted 9-1-1 center(s) 
should take and other appropriate information and 
include the name of the sender and the telephone 
number (business and mobile) at which the sender can 
be reached, and their email address. 



Complete 



Action Plan 



Since March 201 1, Verizon has employed a broadcast email 
process to provide specific ticket information to individual 
PSAPs, and also to provide general information and updates 
on issues that affect multiple PSAPs. 

Verizon has selected a tool for broadcast voice, text and 
email, and is working with 91 1 Directors to establish the 
correct contact lists and process details. Completed for VA 
DCandMD: 10/30/12 

Metropolitan Washington Council of Governments (COG) 91 1 
Steering Committee launched a communications subteam of 
representatives from VZ and each of the 3 COG regions to 
develop processes and thresholds for communications, PSAP 
profiles and shared information preferences. 

Verizon will make every effort to share actionable information 
with PSAPs as soon as we are aware of service interruptions. 
For events that may impact multiple PSAPs, we will 
recommend that voice conference bridges be established to 
enable Verizon to brief PSAPs on the situation and allow for 
questions and discussion. Recommended actions would be 
specific to each PSAP (based on their back-up configuration 
and event impact) and need to be developed jointly between 
Verizon and the PSAP. 



Communications (cont'd) 



Recommendation 




Action Plan 




Verizon work with the jurisdictions to develop, by no 
later than December 31, 2012, a method to semi- 
annually conduct a drill/exercise with each 
jurisdiction on actions to be taken by Verizon and the 
impacted jurisdiction(s) in the event of a potential or 
actual 9-1-1 outage. 


• Verizon engaged the assistance of its Business Continuity 
Emergency Management (BCEM) team to work with 
Verizon's 911 Customer Care Center organization to 
develop and exercise procedures for drills that model 
potential or actual 911 outages with any of the jurisdictions 
that request such a joint exercise. 

• First pilot exercise conducted with City of Norfolk on 
12/04/12. 


Verizon provide the above jurisdictions, during the 
first week of each month, a current contact list; 
beginning with the name and contact information 
(email, business telephone number, business mobile 
telephone number and any other appropriate 
information) for the Verizon account manager 
assigned to the jurisdiction and four immediately 
escalating Verizon personnel up to a Vice President 
level. 


• Based on discussions with the COG 91 1 communications 
subteam, it was decided that escalation lists would be 
provided quarterly (from Verizon to PSAPs), and updated if 
information changed in the interim. 




Complete 












c 



Recommendation 



Verizon, if/when requested by any of the above 
jurisdictions, have a Verizon representative with 
authority to act/react; respond to and to be present 
at the jurisdictions' Emergency Operations Center 
(EOC), to provide current accurate information 
concerning 9-1-1 service and outages, other 
telephone service, etc. and liaison with other parties 
staffing the EOC, when the EOC is activated. 

( ~ A 

Complete 

V . , 



unications (cont'd) 




Verizon has committed to partnering with the Virginia 
Department of Emergency Management and staffing the 
state EOC in Richmond with a Verizon representative upon 
request in the event of an emergency. Staffed the 
Commonwealth EOC for Hurricane Sandy 10/28/12- 
10/31/12. 

Verizon has discussed options for virtual participation in 
any EOC via an "instant messaging - like" application with 
Virginia emergency management leaders. 

Verizon has discussed joint training with Fairfax 
Emergency Management personnel and would welcome 
the opportunity to participate in that activity. Staffed Fairfax 
EOC for Hurricane Sandy 10/29/12 - 10/30/12. 

If PSAP discussions regarding a joint regional 911 EOC 
become the strategy, that would present an excellent 
vehicle for Verizon to be on site in one location serving 
multiple jurisdictions in an emergency situation. 
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FCC CHAIRMAN GENACHOWSKI ANNOUNCES ACTION TO STRENGTHEN 
RELIABILITY AND RESILIENCY OF 9-1-1 COMMUNICATIONS NETWORKS DURING 

MAJOR DISASTERS 

Based on findings and recommendations of a comprehensive inquiry into widespread 9-1-1 service 
failures in the Midwest and mid-Atlantic regions as a result of 2012 derecho storm; 
Final report delivered by the FCC's Public Safety and Homeland Security Bureau 

Washington, D.C. - Federal Communications Commission Chairman Julius Genachowski today 
announced plans to launch a rulemaking to strengthen the reliability and resiliency nationwide of our 
country's 9-1-1 communications networks during major disasters. Widespread outages and disruptions to 
9-1-1 services in the Midwest and mid-Atlantic regions - impacting more than 3.6 million people - led to 
an in-depth FCC inquiry into what went wrong, and what steps should be taken to better ensure public 
safety. The inquiry, conducted by the FCC's Public Safety and Homeland Security Bureau, included in- 
depth investigation, public comment and analysis culminating in a report released today entitled "The 
Impact of the June 2012 Derecho on Communications and Services: Report and Recommendations." 

Chairman Genachowski said, "Americans must be able to reach 9-1-1, especially in times of natural 
disasters. Today's report on the June 2012 derecho finds that a number of preventable system failures 
caused major disruptions to communications providers' networks connecting to 9-1-1 call centers during 
and shortly after the storm. As a result, 9-1-1 was partially or completely unavailable to millions of 
Americans - in some instances, for several days. 

"These failures are unacceptable and the FCC will do whatever is necessary to ensure the reliability of 9- 
1-1. 

"The FCC will soon launch a rulemaking to improve the reliability of existing 9-1-1 networks and prevent 
failures like those outlined in today's report. We will also accelerate the Commission's Next Generation 
(NG) 9-1-1 agenda. NG networks harness the power of the Internet to improve the availability and 
reliability of 9-1-1 communications. 

"Here's the bottom line: We can't prevent disasters from happening, but we can work relentlessly to make 
sure Americans can connect with emergency responders when they need to most." 

About the 2012 Derecho and Impact on Midwest and Mid-Atlantic Regions 

The derecho - a fast-moving, destructive, and deadly storm that developed on June 29, 2012 - caused 
widespread disruptions to communications, especially 9-1-1 services. Shortly after the derecho, Chairman 
Genachowski directed the FCC's Public Safety and Homeland Security Bureau to conduct an inquiry into 
the disruptions, including both the causes of the outages and ways to make the public safer by avoiding 
future outages. 



In the report issued today, the Bureau noted that a significant number of 9-1-1 systems and services were 
partially or completely down for several days after the derecho - from isolated breakdowns in Ohio, New 
Jersey, Maryland, and Indiana to systemic failures in northern Virginia and West Virginia. In all, 
seventy-seven 9-1-1 call centers (known as public safety answering points or "PSAPs") serving more than 
3.6 million people in these six states lost some degree of connectivity, including vital information on the 
location of 9-1-1 calls. Seventeen of the 9-1-1 call centers, mostly in northern Virginia and West 
Virginia, lost service completely, leaving more than 2 million residents unable to reach emergency 
services for varying periods of time. 

Summary of Report Findings 

Unlike hurricanes and superstorms, which are generally well-forecast, derechos are more like 
earthquakes, tornados, and man-made events for which there is little-to-no advance notice and 
opportunity to prepare. As such, the derecho provided a snapshot of the reliability and readiness of a 
portion of the Nation's communications infrastructure in the face of unanticipated disasters — and it 
revealed considerable flaws in the resiliency planning and implementation of the primary 9-1-1 network 
providers in the affected region. In most cases, the disruptions would have been avoided if the 
communications network providers that route calls to 9-1-1 call centers, had fully implemented industry 
best practices and available industry guidance. 

Summary of Report Recommendations 

The Bureau outlined specific suggestions to address the primary causes of the derecho-related outages and 
to promote the reliability and resiliency of 9-1-1 communications networks during disasters. Chief among 
these, the Bureau recommended that the Commission consider actions in the following areas to ensure 
that communications providers: 

Maintain adequate central office backup power 

The Bureau recommended that the Commission consider requiring communications providers to 
maintain robust and reliable backup power at their central offices, supported by appropriate 
testing, maintenance, and records procedures. 

Have reliable network monitoring systems 

The Bureau recommended that the Commission consider requiring providers to take steps to 
ensure that communications providers' monitoring systems are reliable and resilient, and avoid 
cases where a single failure in a monitoring system causes a provider to lose visibility into a 
substantial part of its network. 

Conduct periodic audits of 9-1-1 circuits 

The Bureau recommended that the Commission consider requiring communications providers 
that route calls to 9-1-1 call centers to regularly audit their 9-1-1 circuits and the links that 
transmit location information for 9-1-1 calls. 

Notify 9-1-1 call centers of problems 

The Bureau recommended that the Commission provide more specific guidance, such as the level 
of information that should be included by service providers in their notifications to 9-1-1 call 
centers. 



The Bureau also encouraged the deployment of Next Generation 9-1-1, which offers advantages over 
today's 9-1-1 systems that could have significantly lessened the derecho's impact on emergency 
communications. 

In order to complete today's report, the Bureau conducted an extensive review of confidential outage 
reports, public comments and related documents, as well as interviews of many service providers and 
PSAPs, equipment and backup power vendors, and public safety and community officials. 

As the Bureau was conducting its derecho inquiry, Superstorm Sandy hit the Northeast and Mid-Atlantic 
states. While today's report addresses the most significant communications issues that occurred in the 
wake of the derecho, primarily its devastating impact on the networks that connect 9-1-1 call centers, 
some information gathered during this inquiry also relates to broader network reliability and resiliency 
issues raised during Superstorm Sandy. These topics will be addressed in the Commission's upcoming 
field hearings on the challenges to communications networks during natural disasters and other crises. 

The full Report is available via the below link: 

http://hraunfoss.fcc.aov/edocs public/attachmatch/DOC-318331Al.pdf 
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